{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "## Band Excitation Relaxation Spectroscopy Data Processing\n", "### Jessica Kong, Suhas Somnath, Chris R. Smith, Stephen Jesse\n", "University of Washington
\n", "7/15/2019\n", "\n", "#### This Jupyter Notebook demonstrates how to use Pycroscopy and PyUSID to process Band Excitation relaxation spectra. The code used to configure the notebook, select a file, and preliminary exploration of file parameters (i.e, code before the SHO fits section) was taken from other BE notebooks which were written by Suhas Somnath, Chris R. Smith, and Stephen Jesse.\n", "\n", "![notebook_rules.png](https://raw.githubusercontent.com/pycroscopy/pycroscopy/master/jupyter_notebooks/notebook_rules.png)\n", "\n", "Image courtesy of Jean Bilheux from the [neutron imaging](https://github.com/neutronimaging/python_notebooks) GitHub repository." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Configure the notebook" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "^C\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "Invalid requirement: '#'\n", "\n" ] } ], "source": [ "# Make sure needed packages are installed and up-to-date\n", "import sys\n", "!conda install --yes --prefix {sys.prefix} numpy scipy matplotlib scikit-learn Ipython ipywidgets h5py\n", "!{sys.executable} -m pip install -U --no-deps pycroscopy # this will automatically install pyUSID as well" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", "\n" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "# Ensure python 3 compatibility\n", "from __future__ import division, print_function, absolute_import\n", "\n", "# Import necessary libraries:\n", "# General utilities:\n", "import os\n", "import shutil\n", "\n", "# Computation:\n", "import numpy as np\n", "import h5py\n", "\n", "# Visualization:\n", "# import ipympl\n", "import matplotlib.pyplot as plt\n", "import matplotlib.widgets as mpw\n", "from IPython.display import display, clear_output, HTML\n", "import ipywidgets as widgets\n", "\n", "import pyUSID as usid\n", "# Finally, pycroscopy itself\n", "sys.path.append('..')\n", "import pycroscopy as px\n", "\n", "# Make Notebook take up most of page width\n", "display(HTML(data=\"\"\"\n", "\n", "\"\"\"))" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": [ "# set up notebook to show plots within the notebook\n", "%matplotlib notebook" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Set some basic parameters for computation\n", "This notebook performs some functional fitting whose duration can be substantially decreased by using more memory and CPU cores. We have provided default values below but you may choose to change them if necessary." ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [], "source": [ "max_mem = 1024*8 # Maximum memory to use, in Mbs. Default = 1024\n", "max_cores = None # Number of logical cores to use in fitting. None uses all but 2 available cores." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Make the data pycroscopy compatible\n", "Converting the raw data into a pycroscopy compatible hierarchical data format (HDF or .h5) file gives you access to the fast fitting algorithms and powerful analysis functions within pycroscopy\n", "\n", "#### H5 files:\n", "* are like smart containers that can store matrices with data, folders to organize these datasets, images, metadata like experimental parameters, links or shortcuts to datasets, etc.\n", "* are readily compatible with high-performance computing facilities\n", "* scale very efficiently from few kilobytes to several terabytes\n", "* can be read and modified using any language including Python, Matlab, C/C++, Java, Fortran, Igor Pro, etc.\n", "\n", "#### You can load either of the following:\n", "* Any .mat or .txt parameter file from the original experiment\n", "* A .h5 file generated from the raw data using pycroscopy - skips translation\n", "\n", "You can select desired file type by choosing the second option in the pull down menu on the bottom right of the file window" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Working on:\n", "Z:/Jessica Kong/Data/ORNL Summer-Fall 2018/MeasurementsByKyle/Jess_copy/+6 V Relaxation/BTO_031919_Post+6VPole_RS_+2Vbias_0017.h5\n" ] } ], "source": [ "input_file_path = usid.io_utils.file_dialog(caption='Select translated .h5 file or raw experiment data',\n", " file_filter='Parameters for raw BE data (*.txt *.mat *xls *.xlsx);; \\\n", " Translated file (*.h5)')\n", "\n", "(data_dir, filename) = os.path.split(input_file_path)\n", "\n", "if input_file_path.endswith('.h5'):\n", " # No translation here\n", " h5_path = input_file_path\n", " force = True # Set this to true to force patching of the datafile.\n", " tl = px.io.translators.LabViewH5Patcher()\n", " tl.translate(h5_path, force_patch=force)\n", "else:\n", " # Set the data to be translated\n", " data_path = input_file_path\n", "\n", " (junk, base_name) = os.path.split(data_dir)\n", "\n", "# # Check if the data is in the new or old format. Initialize the correct translator for the format.\n", "# if base_name == 'newdataformat':\n", "# (junk, base_name) = os.path.split(junk)\n", "# translator = px.io.translators.BEPSndfTranslator(max_mem_mb=max_mem)\n", "# else:\n", "# translator = px.io.translators.BEodfTranslator(max_mem_mb=max_mem)\n", "# if base_name.endswith('_d'):\n", "# base_name = base_name[:-2]\n", " # Translate the data\n", " h5_path = translator.translate(data_path, show_plots=True, save_plots=False)\n", "\n", "h5_file = h5py.File(h5_path, 'r+')\n", "print('Working on:\\n' + h5_path)\n", "\n", "h5_main = usid.hdf_utils.find_dataset(h5_file, 'Raw_Data')[0]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "##### Inspect the contents of this h5 data file\n", "The file contents are stored in a tree structure, just like files on a conventional computer.\n", "The data is stored as a 2D matrix (position, spectroscopic value) regardless of the dimensionality of the data. Thus, the positions will be arranged as row0-col0, row0-col1.... row0-colN, row1-col0.... and the data for each position is stored as it was chronologically collected \n", "\n", "The main dataset is always accompanied by four ancillary datasets that explain the position and spectroscopic value of any given element in the dataset." ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Datasets and datagroups within the file:\n", "------------------------------------\n", "/\n", "├ Measurement_000\n", " ---------------\n", " ├ Channel_000\n", " -----------\n", " ├ Bin_FFT\n", " ├ Bin_Frequencies\n", " ├ Bin_Indices\n", " ├ Bin_Step\n", " ├ Excitation_Waveform\n", " ├ Noise_Floor\n", " ├ Position_Indices\n", " ├ Position_Values\n", " ├ Raw_Data\n", " ├ Raw_Data-SHO_Fit_000\n", " --------------------\n", " ├ Guess\n", " ├ Spectroscopic_Indices\n", " ├ Spectroscopic_Values\n", " ├ Raw_Data-SHO_Fit_001\n", " --------------------\n", " ├ Fit\n", " ├ Fit-Double_Exp_000\n", " ------------------\n", " ├ Double_Exp_Fit\n", " ├ Spectroscopic_Indices\n", " ├ Spectroscopic_Values\n", " ├ completed_positions\n", " ├ Guess\n", " ├ Spectroscopic_Indices\n", " ├ Spectroscopic_Values\n", " ├ Spatially_Averaged_Plot_Group_000\n", " ---------------------------------\n", " ├ Bin_Frequencies\n", " ├ Mean_Spectrogram\n", " ├ Spectroscopic_Parameter\n", " ├ Step_Averaged_Response\n", " ├ Spectroscopic_Indices\n", " ├ Spectroscopic_Values\n", "\n", "The main dataset:\n", "------------------------------------\n", "\n", "located at: \n", "\t/Measurement_000/Channel_000/Raw_Data \n", "Data contains: \n", "\tquantity (a.u.) \n", "Data dimensions and original shape: \n", "Position Dimensions: \n", "\tX - size: 10 \n", "\tY - size: 10 \n", "Spectroscopic Dimensions: \n", "\tFrequency - size: 31 \n", "\tPulse_Repeat - size: 250 \n", "\tDC_Offset - size: 2 \n", "\tField - size: 1\n", "Data Type:\n", "\tcomplex64\n", "\n", "The ancillary datasets:\n", "------------------------------------\n", "\n", "\n", "\n", "\n", "\n", "Metadata or attributes in a datagroup\n", "------------------------------------\n", "AFM_InvOLS : 1.1009831191079487e-09\n", "AFM_XLVDT_sensor : 3.8194705242817696e-06\n", "AFM_XPiezo_sensitivity : -1.7495e-07\n", "AFM_YLVDT_sensor : 3.9885994791750315e-06\n", "AFM_YPiezo_sensitivity : 1.8273e-07\n", "AFM_ZLVDT_sensor : 2.43377e-06\n", "AFM_ZPiezo_sensitivity : 2.68351e-08\n", "BE_amplitude_[V] : 1.0\n", "BE_auto_smooth_cond : 1.0\n", "BE_band_width_[Hz] : 100000.0\n", "BE_center_frequency_[Hz] : 384000.0\n", "BE_phase_content : chirp-sinc hybrid\n", "BE_phase_variation : 1.0\n", "BE_pulse_duration_[s] : 0.002\n", "BE_repeats : 4.0\n", "BE_signal_type_ring : 1.0\n", "BE_smoothing : 3208.717049919559\n", "BE_window_adjustment : 0.215625\n", "IO_AFM_platform : Cypher AR14\n", "IO_AI_range : 10.0\n", "IO_AO_range : 10.0\n", "IO_Channel_001_type : none\n", "IO_Channel_002_type : none\n", "IO_Channel_003_type : none\n", "IO_analog_output_amplifier_ring : 1.0\n", "IO_card : 6124\n", "IO_deflection_detector_ring : 0\n", "IO_rate : 4000000.0\n", "IO_sensitivity : 1.0\n", "VS_envelope_max_amp_[V] : 4.0\n", "VS_envelope_min_amp_[V] : 1.0\n", "VS_envelope_shape : 0\n", "VS_mode : VS_Relaxation\n", "VS_num_kernel_repeats : 1\n", "VS_num_meas_per_read_step : 250\n", "VS_num_meas_per_write_step : 5\n", "VS_num_steps_per_envelope : 4\n", "VS_pulse_kernel_polarity : 1\n", "VS_start_kernel_with : 0\n", "VS_step_transition_duration_[s]_ : 2.0\n", "data_type : BERelaxData\n", "grid_current_col : 9\n", "grid_current_row : 9\n", "grid_num_cols : 10\n", "grid_num_rows : 10\n", "grid_set_point_[V] : 1.0\n", "grid_settle_time_[s] : 0.1\n", "grid_transit_time_[s] : 0.1\n", "num_UDVS_bins : 7905\n", "num_UDVS_steps : 255\n", "num_bins : 7905\n", "num_pix : 100\n", "num_steps : 255\n" ] } ], "source": [ "print('Datasets and datagroups within the file:\\n------------------------------------')\n", "usid.hdf_utils.print_tree(h5_file)\n", " \n", "print('\\nThe main dataset:\\n------------------------------------')\n", "print(h5_main)\n", "print('\\nThe ancillary datasets:\\n------------------------------------')\n", "print(h5_main.h5_pos_inds)\n", "print(h5_main.h5_pos_vals)\n", "print(h5_main.h5_spec_inds)\n", "print(h5_main.h5_spec_vals)\n", "\n", "print('\\nMetadata or attributes in a datagroup\\n------------------------------------')\n", "for key, val in usid.hdf_utils.get_attributes(h5_main.parent.parent).items():\n", " print('{} : {}'.format(key, val))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Get some basic parameters from the H5 file\n", "This information will be vital for futher analysis and visualization of the data" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "['X', 'Y'] [10, 10]\n" ] } ], "source": [ "h5_pos_inds = h5_main.h5_pos_inds\n", "pos_dims = h5_main.pos_dim_sizes\n", "pos_labels = h5_main.pos_dim_labels\n", "print(pos_labels, pos_dims)\n", "\n", "parm_dict = h5_main.parent.parent.attrs\n", "is_ckpfm = h5_file.attrs['data_type'] == 'cKPFMData'\n", "if is_ckpfm:\n", " num_write_steps = parm_dict['VS_num_DC_write_steps']\n", " num_read_steps = parm_dict['VS_num_read_steps']\n", " num_fields = 2" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Perform SHO fits to data to extract PFM response " ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Allowed to read 17078 pixels per chunk\n", "Looking at group - Raw_Data-SHO_Fit_000\n", "Looking for new attribute named: num_points\n", "New parm: num_points \t- new parm not in group *****\n", "\n", "Looking at group - Raw_Data-SHO_Fit_001\n", "Looking for new attribute named: num_points\n", "New parm: num_points \t- match: True\n", "Looking for new attribute named: frequencies\n", "New parm: frequencies \t- match: True\n", "Looking for new attribute named: strategy\n", "New parm: strategy \t- match: True\n", "\n", "Returned previously computed results at /Measurement_000/Channel_000/Raw_Data-SHO_Fit_001/Guess\n", "Groups that matched the nomenclature: [, ]\n", "Looking for new attribute named: jac\n", "New parm: jac \t- match: True\n", "Looking for new attribute named: solver_type\n", "New parm: solver_type \t- match: True\n", "Looking for new attribute named: class\n", "New parm: class \t- match: True\n", "Looking for new attribute named: obj_func\n", "New parm: obj_func \t- match: True\n", "Looking for new attribute named: xvals\n", "New parm: xvals \t- match: True\n", "\n", "Returned previously computed results at /Measurement_000/Channel_000/Raw_Data-SHO_Fit_001/Fit\n" ] } ], "source": [ "sho_fit_points = 5 # The number of data points at each step to use when fitting\n", "sho_override = False # Force recompute if True\n", "\n", "h5_main_SHO_fitter = px.analysis.BESHOfitter(h5_main, parallel=True, verbose=True)\n", "h5_main_SHO_guess = h5_main_SHO_fitter.do_guess(strategy='complex_gaussian', options={'num_points':sho_fit_points},\n", " processors=max_cores, max_mem=max_mem, override=sho_override)\n", "h5_main_SHO_fit = h5_main_SHO_fitter.do_fit(processors=max_cores, max_mem=max_mem, override=sho_override)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Perform functional fits to PFM relaxation data " ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [], "source": [ "starts_with = 'write'\n", "#experimentally obtained tip sensitivity \n", "tip_sens = 83.53 * 1e3 #pm/V \n", "#phase offset to get raw phase centered at 0. can obtain by plotting: \n", "# plt.hist(np.concatenate(h5_main_SHO_fit['Phase [rad]'].reshape(-1,1)), bins=1000);\n", "phase = 3.0\n", "#fit method, can be 'Exponential' 'Double_Exp' or 'Logistic'\n", "fit_method = 'Double_Exp'" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Consider calling test() to check results before calling compute() which computes on the entire dataset and writes back to the HDF5 file\n" ] } ], "source": [ "decayfit = px.analysis.BERelaxFit(h5_main_SHO_fit, sens=tip_sens, phase_off=phase, starts_with=starts_with, fit_method=fit_method)\n", "#start and end positions should be 0, and spatial dimension of h5_main; currently is user specified because it has yet to be integrated to the BERelaxFit process class.\n", "decayfit._start_pos = 0\n", "decayfit._end_pos = h5_main.shape[0]\n", "decayfit._cores = 1\n", "#decayfit.compute()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Visualize the data" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "98f8de68119141e6b8e0bfc7e56e3688", "version_major": 2, "version_minor": 0 }, "text/plain": [ "interactive(children=(IntSlider(value=0, continuous_update=False, description='App. Voltage', max=0), IntSlide…" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "\n", "h5_main_fit = h5_file['Measurement_000/Channel_000/Raw_Data-SHO_Fit_001/Fit-Double_Exp_000/Double_Exp_Fit']\n", "#make widget sliders for x-, and y-axes, voltage, and time \n", "slide_x = widgets.IntSlider(min=0, max=h5_main_SHO_fit.pos_dim_sizes[0] - 1, description='Col')\n", "slide_y = widgets.IntSlider(min=0, max=h5_main_SHO_fit.pos_dim_sizes[1] - 1, description='Row')\n", "slide_v = widgets.IntSlider(min=0, max=decayfit.no_rs_spectra-1, step=1, description='App. Voltage',\n", " continuous_update=False, value = 0)\n", "slide_t = widgets.IntSlider(min=0, max=decayfit.no_time_steps - 1, description='Time Slice',\n", " continuous_update=False)\n", "#make plotting interactive with widgets\n", "widgets.interact(px.viz.be_viz_utils.viz_berelaxfit, berelaxfit = widgets.fixed(h5_main_fit), t_time=slide_t, x_col=slide_x, h_row=slide_y, \n", " bias_ind=slide_v, sensitivity=widgets.fixed(tip_sens), phase_offset=widgets.fixed(phase), fit_method=widgets.fixed(fit_method), starts_with = widgets.fixed(starts_with));" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.3" } }, "nbformat": 4, "nbformat_minor": 2 }